Representing Narrative in Multimedia Systems

This report investigates the hypothesis: patterns in multimedia discourses can be used to specify and extract machine-executable representations of media content that capture important elements of stories. Multimedia Information Systems are amongst the more pervasive of information systems. The physical and conceptual task of synthesising two or more modalities of communication-visual and linguistic, to produce information, to convey 'emotions' and to convey instruction is a complex one. The converse task of extracting information, identifying emotional states and locating instruction is perhaps even more complex. This report considers the structure of multimedia data: we consider multimedia data as having an underlying story or narrative. We are synthesising notions from narrative theory, corpus linguistics and Language for Special Purpose (LSP) and case studies are considered for these notions. Firstly, a graphical display of narrative theory and narrative structural phenomena is illustrated as a case study for modelling narrative, and a notation for dealing with 'simple' narrative elements (events, states and existents) is developed. Then, random film collateral texts – Audio Description and Screenplay film scripts-are analysed in terms of mental states or emotions to see whether these states are indicatory of narrative elements and whether they can be extracted automatically. We then try to extract repeating lexical patterns from films to examine whether we can develop a local grammar for film using two film script corpora as a case study. A method is discussed for extracting qualitative information from linguistic symbols. Informed by ideas of special language, where trained experts communicate their needs in a specialised vocabulary, film collateral texts (film scripts) are analysed in terms of established corpus linguistic analysis methods to elicit information about frequent words and their collocates. From this analysis evidence of local grammars emerge as well as repeating surface patterns indicating what is believed to be important narrative elements. We believe that further study of the corpora will reveal a local grammar for the collateral text and cues for important natural elements that can be used to populate a machine-executable representation of multimedia data.